Complete Agent-driven Model-based System Testing for Autonomous Systems

In this position paper, a novel approach to testing complex autonomous transportation systems (ATS) in the automotive, avionic, and railway domains is described. It is intended to mitigate some of the most critical problems regarding verification and validation (V&V) effort for ATS. V&V is known to become infeasible for complex ATS, when using conventional methods only. The approach advocated here uses complete testing methods on the module level, because these establish formal proofs for the logical correctness of the software. Having established logical correctness, system-level tests are performed in simulated cloud environments and on the target system. To give evidence that 'sufficiently many' system tests have been performed with the target system, a formally justified coverage criterion is introduced. To optimise the execution of very large system test suites, we advocate an online testing approach where multiple tests are executed in parallel, and test steps are identified on-the-fly. The coordination and optimisation of these executions is achieved by an agent-based approach. Each aspect of the testing approach advocated here is shown to either be consistent with existing standards for development and V&V of safety-critical transportation systems, or it is justified why it should become acceptable in future revisions of the applicable standards.Comment: In Proceedings FMAS 2021, arXiv:2110.1152

An Agency-Directed Approach to Test Generation for Simulation-based Autonomous Vehicle Verification

Simulation-based verification is beneficial for assessing otherwise dangerous or costly on-road testing of autonomous vehicles (AV). This paper addresses the challenge of efficiently generating effective tests for simulation-based AV verification using software testing agents. The multi-agent system (MAS) programming paradigm offers rational agency, causality and strategic planning between multiple agents. We exploit these aspects for test generation, focusing in particular on the generation of tests that trigger the precondition of an assertion. On the example of a key assertion we show that, by encoding a variety of different behaviours respondent to the agent's perceptions of the test environment, the agency-directed approach generates twice as many effective tests than pseudo-random test generation, while being both efficient and robust. Moreover, agents can be encoded to behave naturally without compromising the effectiveness of test generation. Our results suggest that generating tests using agency-directed testing significantly improves upon random and simultaneously provides more realistic driving scenarios.Comment: 18 pages, 8 figure

Risk-Based Triggering of Bio-inspired Self-preservation to Protect Robots from Threats

Safety in autonomous systems has been mostly studied from a human-centered perspective. Besides the loads they may carry, autonomous systems are also valuable property, and self-preservation mechanisms are needed to protect them in the presence of external threats, including malicious robots and antagonistic humans. We present a biologically inspired risk-based triggering mechanism to initiate self-preservation strategies. This mechanism considers environmental and internal system factors to measure the overall risk at any moment in time, to decide whether behaviours such as fleeing or hiding are necessary, or whether the system should continue on its task. We integrated our risk-based triggering mechanism into a delivery rover that is being attacked by a drone and evaluated its effectiveness through systematic testing in a simulated environment in Robot Operating System (ROS) and Gazebo, with a variety of different randomly generated conditions. We compared the use of the triggering mechanism and different configurations of self-preservation behaviours to not having any of these. Our results show that triggering self-preservation increases the distance between the drone and the rover for many of these configurations, and, in some instances, the drone does not catch up with the rover. Our study demonstrates the benefits of embedding risk awareness and self-preservation into autonomous systems to increase their robustness, and the value of using bio-inspired engineering to find solutions in this area